Single image upscaling, or image super-resolution (SR), may use systems and techniques to obtain a high-resolution (HR) image from a low-resolution (LR) image. Issues may occur when upscaling a LR image with potential noise corruption, such as Joint Photographic Experts Group (JPEG) compression artifacts, sensor noise and/or other noise. Various noises are inevitable in digital images. Some noises are not perceivable in the original image resolution, but the noises may become strong artifacts after the images are upscaled using general upscaling techniques such that the noise is perceivable in the upscaled HR image. Upscaling techniques designed for noise-free images may have challenges distinguishing between noise and signal in image data. One approach to deal with this problem is to apply a de-noising operation as a first process to reduce the noise level in the low-resolution (LR) image, and then to upscale the image in a separate second process so that the result high-resolution (HR) image has less artifacts. A de-noising process is a process to reduce the noise in the image, where noise may generally refer to noise corruption as discussed above. One consequence of this two-step process is that any de-noising operation will also smooth out some original signal details, causing more blur in the resulting HR image.
More recently, neural networks, including deep neural networks, have been used in single image upscaling. Despite using neural networks in single image upscaling, challenges and problems remain, including the ones described above with respect to noise present in the original image resolution. As above, some upscaling techniques using neural networks treat the de-noising and upscaling processes as two independent processes and combine the results from the two processes in an ad-hoc manner.
In general, neural networks, especially deep neural networks have been very successful in modeling high-level abstractions in data. Neural networks are computational models used in machine learning made up of nodes organized in layers. The nodes are also referred to as artificial neurons, or just neurons, and perform a function on provided input to produce some output value. A neural network requires a training period to learn the parameters, i.e., weights, used to map the input to a desired output. The mapping occurs via the function. Thus, the weights are weights for the mapping function of the neural network.
Each neural network is trained for a specific task, e.g., image upscaling, prediction, classification, encoding, etc. The task performed by the neural network is determined by the inputs provided, the mapping function, and the desired output. Training can either be supervised or unsupervised. In supervised training, training examples are provided to the neural network. A training example includes the inputs and a desired output. Training examples are also referred to as labeled data because the input is labeled with the desired output. The network learns the values for the weights used in the mapping function that most often result in the desired output when given the inputs. In unsupervised training, the network learns to identify a structure or pattern in the provided input. In other words, the network identifies implicit relationships in the data. Unsupervised training is used in deep neural networks as well as other neural networks and typically requires a large set of unlabeled data and a longer training period. Once the training period completes, the neural network can be used to perform the task it was trained for.
In a neural network, the neurons are organized into layers. A neuron in an input layer receives the input from an external source. A neuron in a hidden layer receives input from one or more neurons in a previous layer and provides output to one or more neurons in a subsequent layer. A neuron in an output layer provides the output value. What the output value represents depends on what task the network is trained to perform. Some neural networks predict a value given in the input. Some neural networks provide a classification given the input. When the nodes of a neural network provide their output to every node in the next layer, the neural network is said to be fully connected. When the neurons of a neural network provide their output to only some of the neurons in the next layer, the network is said to be convolutional. In general, the number of hidden layers in a neural network varies between one and the number of inputs.
To provide the output given the input, the neural network must be trained, which involves learning the proper value for a large number (e.g., millions) of parameters for the mapping function. The parameters are also commonly referred to as weights as they are used to weight terms in the mapping function. This training is an iterative process, with the values of the weights being tweaked over thousands of rounds of training until arriving at the optimal, or most accurate, values. In the context of neural networks, the parameters are initialized, often with random values, and a training optimizer iteratively updates the parameters, also referred to as weights, of the network to minimize error in the mapping function. In other words, during each round, or step, of iterative training the network updates the values of the parameters so that the values of the parameters eventually converge on the optimal values.